Image processing apparatus, method for controlling the same, and storage medium

ABSTRACT

A camera scanner captures an image of a document stand on which an object is placed using a camera unit. The camera scanner obtains an IP address of an electronic terminal (a smartphone) and an identifier of an image from the image of a display screen of the electronic terminal placed on the document stand by detecting and decoding a QR code. The camera scanner is connected to the electronic terminal based on the IP address, transmits a data transfer request including the identifier of the image to the electronic terminal, and receives image data transferred from the electronic terminal in response to the data transfer request. Then the camera scanner projects and plays an image generated based on the data transferred from the electronic terminal on the document stand using a projector.

BACKGROUND Field of the Disclosure

The present disclosure relates to an image processing apparatus whichperforms display based on data transferred from an electronic terminal,a method for controlling the image processing apparatus, and a storagemedium.

Description of the Related Art

In general, a technique associated with a projector and a camera scannerwhich has a user interface and which uses a camera and a distance imagesensor has been disclosed. For example, in a technique disclosed inJapanese Patent Laid-Open No. 2015-212898, a user interface is displayedon a real object, such as a sheet, in an overlapping manner byprojection display performed by a projector. Therefore, a user may usethe real object as an interface with electronic data.

According to such a technique, in an operation using both a paperdocument and electronic data, improvement of operation efficiency isexpected. In a case where this technique is applied to support anoperation, performed by a salesperson, of making a description for acustomer using paper (a face-to-face operation), when the salespersontraces a portion to be emphasized on a physical paper document for thecustomer with a finger, a red line, for example, may be displayed in anoverlapping manner. In general, a sheet is required to be scanned oncebefore being computerized, and therefore, effective support of theface-to-face operation is realized when this technique is used.

However, the technique described above only simulates an operationtarget which does not include electronic data, such as paper. Here, ifan image based on data displayed in an electronic terminal, such as atablet terminal or a smartphone, is projected and displayed by aprojector of a camera scanner, support of the face-to-face operationusing a larger screen may become available. However, in the techniquedescribed above, projection and display of data displayed in theelectronic terminal may not be easily performed by the projector of thecamera scanner.

SUMMARY

According to some embodiments, an image processing apparatus includes animaging unit configured to capture a document stand on which an objectis placed, a memory device configured to store a set of instructions,and at least one processor configured to execute instructions to detectan encoded image generated by a predetermined encoding method in animage of a display screen of an electronic terminal placed on thedocument stand which is captured by the imaging unit, obtain connectioninformation for connection to the electronic terminal and an identifierof data transferred from the electronic terminal by performing adecoding process on the detected encoded image, make a connection to theelectronic terminal using the connection information, transmit a datatransfer request including the identifier to the electronic terminal,receive data transferred from the electronic terminal in response to thedata transfer request, and project an image generated based on the datatransferred from the electronic terminal on the document stand bycontrolling a projection unit.

Further features of various embodiments will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically illustrating a network configurationof a camera scanner.

FIGS. 2A to 2C are diagrams schematically illustrating the appearance ofthe camera scanner.

FIG. 3 is a diagram illustrating a hardware configuration of acontroller unit of the camera scanner.

FIGS. 4A and 4B are diagrams illustrating a functional blockconfiguration and a control sequence of the camera scanner,respectively.

FIGS. 5A to 5D are flowcharts of processes executed by an objectdetection unit.

FIG. 6 is a diagram illustrating a hardware configuration of acontroller unit of a smartphone.

FIGS. 7A to 7G are diagrams illustrating screen transitions of thesmartphone at a time of a data transfer process.

FIGS. 8A to 8H are diagrams illustrating states of a surface D adocument stand according to a first embodiment.

FIGS. 9A and 9B are flowcharts of transfer processes of the smartphoneand the camera scanner, respectively.

FIGS. 10A to 10K are diagrams illustrating states of a surface of adocument stand according to a second embodiment.

FIG. 11 is a flowchart of a transfer process performed by a camerascanner according to the second embodiment.

FIG. 12 is a flowchart of a reception data display region determinationprocess.

FIGS. 13A to 13E are diagrams illustrating the determination ofreception data display regions.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be describedwith reference to the accompanying drawings.

First Embodiment

FIG. 1 is a diagram illustrating a network configuration of a camerascanner 101 according to a first embodiment.

As illustrated in FIG. 1, the camera scanner 101 includes a hostcomputer 102, a printer 103, and a wireless access point 105 through anetwork 104, such as Ethernet. Furthermore, the camera scanner 101 isconnectable to a smartphone 106, which is an example of an electronicterminal, through the wireless access point 105. In the networkconfiguration of FIG. 1, a scan function of the camera scanner 101 ofreading an image in response to an instruction issued by the hostcomputer 102 and a print function of the printer 103 of outputting scandata obtained by the reading may be executed. Furthermore, in thenetwork configuration of FIG. 1, the scan function and the printfunction may be executed by issuing a direct, instruction to the camerascanner 101 without using the host computer 102. Furthermore, the camerascanner 101 and the smartphone 106 may perform socket communication witheach other in accordance with TCP/IP.

Configuration of Camera Scanner

FIGS. 2A and 2B are diagrams schematically illustrating a configurationof the camera scanner 101, which is an example of a data transferapparatus in this embodiment.

As illustrated in FIG. 2A, the camera scanner 101 includes a controllerunit 201, a camera unit 202, an arm unit 203, a short focus projector207 (hereinafter referred to as a “projector 207”), and a range imagesensor unit 208. The controller unit 201, which is a main body of thecamera scanner 101; the camera unit 202, which performs imaging; theprojector 207; and the range image sensor unit 208 are connected to oneanother through the arm unit 203. The arm unit 203 may be bent andstretched using a joint.

Furthermore, a document stand 204 on which the camera scanner 101 ismounted is also illustrated in FIG. 2A. A lens of the camera unit 202and a lens of the range image sensor unit 208 are directed to thedocument stand 204. The camera unit 202 captures an image of a readingregion 205 surrounded by a dotted line in FIG. 2A so as to read an imageof an object, a document, or the like placed on the reading region 205.The range image sensor unit 208 projects a certain pattern image in thereading region 205 and captures the pattern image so as to obtain arange image. In the example of FIG. 2A, a document 206 is included inthe reading region 205, and therefore, the camera scanner 101 may readthe document 206. Furthermore, a turntable 209 is disposed on thedocument stand 204. The turntable 209 is rotatable in accordance with aninstruction issued by the controller unit 201. An object placed on theturntable 209 may be rotated by rotating the turntable 209, as viewedfrom the camera unit 202.

Note that, although the camera unit 202 may capture an image in only asingle resolution, the camera unit 202 may be capable of performinghigh-resolution image capturing and low-resolution image capturing.

Furthermore, although not illustrated in FIGS. 2A to 2C, the camerascanner 101 may further include a liquid crystal display (LCD) touchpanel 330, a speaker 340, and an external memory 350 illustrated in FIG.3.

In FIG. 2B, in addition to the configuration of the camera scanner 101,coordinate systems of the camera scanner 101 are also illustrated. Inthe camera scanner 101, a camera coordinate system 221 of the cameraunit 202, a range image sensor coordinate system 220 of the range imagesensor unit 208, and a projector coordinate system 222 of the projector207 are individually defined. In the coordinate systems 220 to 222, animage plane captured by the camera unit 202; an image plane captured byan RGB (red, green, and blue) camera 363 of FIG. 3, which is describedbelow and is included in the range image sensor unit 208; and an imageplane projected by the projector 207 are defined as XY planes.Furthermore, in each of the coordinate systems 220 to 222, a directionwhich is orthogonal to the XY planes (the image planes) is defined as aZ direction. Furthermore, in the camera scanner 101, an orthogonalcoordinate system 223, having a plane including the document stand 204,which is set as an XY plane, and having a direction vertical to the XYplane (upward from the XY plan), which is set as a Z axis, is defined sothat individual three-dimensional (3D) data of the coordinate systems220 to 222 is integrally used.

Here, FIG. 2C illustrates the relationship between a space representedby the orthogonal coordinate system 223 and the camera coordinate system221 including the camera unit 202 at a center and a camera image plane224 captured by the camera unit 202 as an example case of conversion ofthe coordinate systems. A 3D point P[X, Y, Z] in the orthogonalcoordinate system 223 may be converted into a 3D point Pc[Xc, Yc, Zc] inthe camera coordinate system 221 in accordance with Expression (1):[Xc,Yc,Zc]^(T)=[Rc|tc][X,Y,Z,1]^(T).  (1)

Here, “RC” and “tc” in Expression (1) are external parameters obtainedin accordance with an orientation (rotation) and a position(translation) of the camera unit 202 relative to the orthogonalcoordinate system 223, and “Rc” is referred to as a rotation matrix and“tc” is referred to as a translation vector hereinafter. On the otherhand, a 3D point defined by the camera coordinate system 221 may beconverted into the orthogonal coordinate system 223 in accordance withExpression (2) below:[X,Y,Z]^(T)=[Rc ⁻¹ |−R ⁻¹ tc][X,Y,Z,1]^(T).  (2)

Furthermore, the two-dimensional (2D) camera image plane 224 captured bythe camera unit 202 is obtained when the camera unit. 202 converts 3Dinformation in a 3D space into 2D information. Specifically, the cameraimage plane 224 converts a 3D point Pc[Xc, Yc, Zc] on the cameracoordinate system 221 into a 2D coordinate pc[xp, yp] by perspectiveprojection in accordance with Expression (3) below:λ[xp,yp,1]^(T) =A[Xc,Yc,Zc]^(T).  (3)

Here, “A” in Expression (3) is referred to as an internal parameter ofthe camera unit 202 and is a matrix of 3 rows by 3 columns representedby a focus distance, an image center, and the like.

As described above, in accordance with Expressions (1) and (3), a 3Dpoint group represented by the orthogonal coordinate system 223 isconverted into a 3D point group coordinate in the camera coordinatesystem 221 or the camera image plane 224. Note that internal parametersof the various hardware devices, such as the camera unit 202, the rangeimage sensor unit 208, and the projector 207, and position andorientation (an external parameter) relative to the orthogonalcoordinate system 223, are calibrated in advance by a generalcalibration method. Hereinafter, the term “3D point group” indicates 3Ddata in the orthogonal coordinate system 223 unless otherwise noted.

Hardware Configuration of Controller Unit of Camera Scanner

FIG. 3 is a diagram illustrating a hardware configuration of thecontroller unit 201 of the camera scanner 101.

The controller unit 201 of FIG. 3 includes a central processing unit(CPU) 302, a random access memory (RAM) 303, a read only memory (ROM)304, a hard disk drive (HDD) 305, and a network interface (I/F) 306. Thecontroller unit 201 further includes an image processor 307, a cameraI/F 308, a display controller 309, a serial I/F 310, an audio controller311, and a universal serial bus (USB) controller 312. The various unitsare connected to one another through a system bus 301.

The CPU 302 controls all the operations of the controller unit 201. TheRAM 303 is a volatile memory. The ROM 304 is a nonvolatile memory whichstores a boot program of the CPU 302. The HDD 305 has capacity largerthan that of the RAM 303. The HDD 305 stores a control program of thecamera scanner 101 to be executed by the controller unit 201.

The CPU 302 executes the boot program stored in the ROM 304 at a time ofactivation, such as power-ON. When the boot program is executed, thecontrol program stored in the HDD 305 is read and developed in the RAM303. After executing the boot program, the CPU 302 subsequently executesthe control program developed in the RAM 303 so as to control thevarious units. Furthermore, the CPU 302 stores data to be used for anoperation based on the control program in the RAM 303 and performsreading and writing. The HDD 305 may further store various types ofsetting data required for the operation based on the control program andimage data generated by camera inputs which are to be read and writtenunder control of the CPU 302. The CPU 302 performs communication withother devices on the network 104 of FIG. 1 through the network I/F 306.

The image processor 307 reads and processes the image data stored in theRAM 303 and rewrites the image data in the RAM 303. Note that examplesof the image processing performed by the image processor 307 includerotation, magnification, and color conversion.

The camera I/F 308 is connected to the camera unit 202 and the rangeimage sensor unit 208, obtains image data from the camera unit 202 inaccordance with an instruction issued by the CPU 302, obtains rangeimage data from the range image sensor unit 208, and writes the data inthe RAM 303. The camera I/F 308 transmits a control command suppliedfrom the CPU 302 to the camera unit 202 and the range image sensor unit208 so as to perform a setting of the camera unit 202 and the rangeimage sensor unit 208.

The display controller 309 is connected to the projector 207 and the LCDtouch panel 330. The display controller 309 controls the images that aredisplayed on a display of the projector 207 and the LCD touch panel 330in accordance with an instruction issued by the CPU 302.

The serial I/F 310 inputs and outputs a serial signal. The serial I/F310 is connected to the turntable 209 and transmits an instruction forstarting rotation, an instruction for terminating rotation, and aninstruction for indicating a rotation angle supplied from the CPU 302 tothe turntable 209. The serial I/F 310 is also connected to the LCD touchpanel 330. When the LCD touch panel 330 is touched, for example, the CPU302 obtains a coordinate of a touched position, for example, through theserial I/F 310. The audio controller 311 is connected to the speaker340, converts audio data into an analog audio signal in accordance withan instruction issued by the CPU 302, and outputs audio signals throughthe speaker 340.

The USB controller 312 controls an external USB device in accordancewith an instruction issued by the CPU 302. Here, the USB controller 312is connected to the external memory 350, such as a USB memory or an SDcard, and performs reading of data from and writing of data to theexternal memory 350.

Functional Configuration of Control Program of Camera Scanner

FIG. 4A is a diagram illustrating a functional block configuration 401formed when the CPU 302 executes the control program of the camerascanner 101. FIG. 4B is a sequence diagram of a flow of processesperformed by various units (e.g., software modules) of the functionalblock configuration 401 under control of a main controller 402. Notethat the control program for the camera scanner 101 is stored in the HDD305 as described above and is executed by the CPU 302 after beingdeveloped in the RAM 303 at a time of activation.

The main controller 402 serves as a control center and controls thevarious units included in the functional block configuration 401 inaccordance with a flow illustrated in the sequence diagram of FIG. 4B.Specifically, processes of the units described below are individuallycontrolled by the main controller 402 in accordance with the flow of thesequence of FIG. 4B.

An image obtaining unit 416 is a module which performs an image inputprocess and includes a camera image obtaining unit 407 and a range imageobtaining unit 408. The camera image obtaining unit 407 obtains imagedata output by the camera unit 202 through the camera I/F 308 and storesthe image data in the RAM 303. The range image obtaining unit 408obtains range image data output by the range image sensor unit 208through the camera I/F 308 and stores the range image data in the RAM303.

A recognition processing unit 417 detects and recognizes a movement ofan object on the document stand 204 based on the image data obtained bythe camera image obtaining unit 407 and the range image data obtained bythe range image obtaining unit 408. The recognition processing unit 417includes a gesture recognition unit 409 and an object detection unit410. Although not illustrated, the recognition processing unit 417 mayfurther include a document region extraction unit, a document regionconversion unit, a feature point extraction unit, a 2D image documentcontour calculation unit, a distance calculation unit, and a documentcontour calculation unit. The document region extraction unit extracts adocument region from image data. The document region conversion unitperforms coordinate conversion and the like on the document region. Thefeature point extraction unit extracts a feature point of an image ofthe document region. The 2D image document contour calculation unitcalculates a contour of the 2D image in the document region. Thedistance calculation unit calculates a distance between feature pointscorresponding to each other. The document contour calculation unitcalculates a contour of the document region. The gesture recognitionunit 409 constantly obtains images on the document stand 204 from theimage obtaining unit 416. When detecting a gesture, such as a touch by auser, the gesture recognition unit 409 notifies the main controller 402of the gesture. When receiving a notification of an object placementwaiting process or an object removal waiting process from the maincontroller 402, the object detection unit 410 obtains an image obtainedby capturing the document stand 204 from the image obtaining unit 416.The object detection unit 410 performs a process of detecting a timewhen an object is placed on the document stand 204 and becomesstationary or a time when the object is removed. The process of theobject detection unit 410 included in the recognition processing unit417 will be described in detail hereinafter with reference to FIGS. 5Ato 5D.

A scan processing unit 418 is a module which actually performs scanningon a target and includes a plane document image capturing unit 411, abook image capturing unit 412, and a solid shape measurement unit 413.The plane document image capturing unit 411, the book image capturingunit 412, and the solid shape measurement unit 413 perform processessuitable for a plane document, a book, and a solid, respectively, andoutput data of corresponding formats.

A user interface unit 403 includes a graphical user interface (GUI)component generation display unit 414 and a projection region detectionunit 415. When receiving a request from the main controller 402, the GUIcomponent generation display unit 414 generates GUI components includinga message and a button. Then the GUI component generation display unit414 requests a display unit 406 to display the generated GUI components.By this, the GUI components (e.g., user interface images) are displayedon the document stand 204 or an object mounted on the document stand 204in an overlapping manner. Accordingly, in this case, the user may use areal object on the document stand 204 as an interface with electronicdata. Note that a display region of the GUI components on the documentstand 204 is detected by the projection region detection unit 415.

The display unit 406 displays the requested GUI components on theprojector 207 or the LCD touch panel 330 through the display controller309. Since the projector 207 faces the document stand 204, the GUIcomponents may be projected on the document stand 204 for display.

Furthermore, the user interface unit 403 receives a gesture operation,such as a touch operation recognized by the gesture recognition unit 409or an input operation performed using the LCD touch panel 330 throughthe serial I/F 310, and a coordinate of the operation. Thereafter, theuser interface unit 403 determines the content of an operation (such aspressing of a button) performed by the user by matching the content ofan operation screen that is being rendered and an operation coordinate.The main controller 402 accepts the user operation when receiving theoperation content.

A network communication unit 404 performs communication in accordancewith TCP/IP with other devices on the network 104 through the networkI/F 306.

A data management unit 405 stores various data, including work datagenerated by execution of the control program in a certain region of theHDD 305, and manages the data. Examples of the data include scan datagenerated by the plane document image capturing unit 411, the book imagecapturing unit 412, and the solid shape measurement unit 413.

A data transfer processing unit 420 performs a process of data transferwith the smartphone 106 connected through the network communication unit404. The data transfer process will be described in detail hereinafter.

Range Image Sensor and Range Image Obtaining Unit

The range image sensor unit 208 employs a pattern projection methodusing infrared light and includes an infrared pattern projection unit361, an infrared camera 362, and the RGB camera 363 as illustrated inFIG. 3. The infrared pattern projection unit 361 projects a 3Dmeasurement pattern invisible infrared light on a target object. Theinfrared camera 362 reads the 3D measurement pattern projected on thetarget object. The RGB camera 363 captures an image of visible lightusing an RGB signal.

Although the infrared pattern projection method employed in the rangeimage sensor unit 208 as described above in this embodiment, a rangeimage sensor employing another method may be used. For example, a stereomethod for performing stereoscopic viewing using two RGB cameras or aTime-of-Flight (TOF) method for performing range measurement bydetecting a flight time of laser light may be employed.

Gesture Recognition Unit

A process performed by the gesture recognition unit 409 will bedescribed in detail.

The gesture recognition unit 409 obtains a range image and a 3D pointgroup from the range image obtaining unit 408. Subsequently, the gesturerecognition unit 409 extracts a 3D point group of a hand by extracting a3D point group of a skin color that is located in a position equal to orhigher than a predetermined height relative to a plane including thedocument stand 204. Thereafter, the gesture recognition unit 409generates a 2D image by projecting the extracted 3D point group of thehand on the plane of the document stand 204 and detects an outline ofthe hand. Here, the correspondence relationships between points of the3D point group of the hand and coordinates of the 2D image projected onthe plane of the document stand 204 are also stored. Subsequently, thegesture recognition unit 409 calculates curvatures at points on thedetected outline of the hand and detects points having the calculatedcurvatures smaller than a predetermined value as a fingertip.Thereafter, the gesture recognition unit 409 calculates a distancebetween the detected fingertip and the plane including the documentstand 204 using the 3D point group in accordance with the correspondencerelationships. When the calculated distance is equal to or smaller thana small predetermined value, the gesture recognition unit 409 detects atouch gesture and notifies the main controller 402 of the detection.

Although a method for detecting a fingertip in a 2D image projectedusing a 3D point group in fingertip detection has been described, atarget image for the fingertip detection is not limited to this. Forexample, a background difference of a range image may be extracted, or aregion corresponding to a hand may be extracted from a skin color regionof an RGB image, and thereafter, a region corresponding to a fingertipmay be detected in the hand region by the method described above(outline curvature calculation or the like).

Process of Object Detection Unit

Hereinafter, a process performed by the object detection unit 410 willbe described with reference to flowcharts of FIGS. 5A to 5D. Note that,in a description below, step S801 to step S834 in the processes in theflowcharts of FIGS. 5A to 5D are simply referred to as S801 to S834.This is true of other flowcharts described hereinafter.

When the process of the flowchart of FIG. 5A is started, the objectdetection unit 410 performs an initializing process in S801 thatincludes S811 to S813. In S811, the object detection unit 410 obtains acamera image and a range image for one frame from the camera imageobtaining unit 407 and the range image obtaining unit 408, respectively.In S812, the object detection unit 410 stores the obtained camera imageas a preceding-frame camera image. In S813, the object detection unit410 stores the obtained camera image and the obtained range image as adocument stand background camera image and a document stand backgroundrange image, respectively. Hereinafter, the terms “document standbackground camera image” and “document stand background range image”indicate the camera image and the range image obtained in this step.After S813, the object detection unit 410 proceeds to S802.

In S802, the object detection unit 410 determines whether an object hasbeen placed on the document stand 204 (an object placement detectionprocess). This process will be described in detail below with referenceto FIGS. 5B and 5C. After S802, the object detection unit 410 proceedsto S803.

In S803, the object detection unit 410 determines whether the objectthat has detected on the document stand 204 in S802 has been removed (anobject removal detection process). This process will be described indetail below with reference to FIG. 5D. After S803, the object detectionunit 410 terminates the process in the flowchart of FIG. 5A.

FIG. 5B is a flowchart of the object placement detection process in S802of FIG. 5A performed when a first object is placed in detail. Thisprocess is performed after the initializing process in FIG. 5A and theobject removal detection process in FIG. 5D described below.

In S821 of FIG. 5B, the object detection unit 410 obtains a camera imageand a range image for one frame from the camera image obtaining unit 407and the range image obtaining unit 408, respectively. After S821, theobject detection unit 410 proceeds to S822.

In S822, the object detection unit 410 calculates a difference betweenthe obtained range image and the preceding-frame range image andcalculates a difference value by adding absolute values so as to obtaina range image in a range to the document stand (a document standbackground range image). After S822, the object detection unit 410proceeds to S823.

In S823, the object detection unit 410 determines whether the calculateddifference value is equal to or larger than a predetermined value. Whenthe determination is negative in S823 (NO), the object detection unit410 determines that no object is placed on the document stand 204 andproceeds to S828. In S828, the object detection unit 410 stores thecamera image and the range image in the current frame as apreceding-frame camera image and a preceding-frame range image,respectively, before returning to S821. On the other hand, when thedetermination is affirmative in S823 (YES), the process proceeds toS824. In S824, the object detection unit 410 calculates a differencevalue between the range image obtained in S821 and the preceding-framerange image similarly to S822. After S824, the object detection unit 410proceeds to S825.

In S825, the object detection unit 410 determines whether the calculateddifference value is equal to or smaller than a predetermined value. Whenthe determination is negative in S825 (NO), the object detection unit410 determines that an object on the document stand 204 is moving andproceeds to S828. In S828, the object detection unit 410 stores therange image in the current frame as a preceding-frame range image beforereturning to S821. On the other hand, when the determination isaffirmative in S825 (YES), the object detection unit 410 proceeds toS826.

In S826, the object detection unit 410 determines whether a state inwhich the difference value is equal to or smaller than a predeterminedvalue, that is, a stationary state of the object on the document stand204, has continued for a predetermined number of frames in accordancewith the number of consecutive positive results of the determination inS825. When the determination is negative in S826 (NO), the objectdetection unit 410 proceeds to S828. In S828, the object detection unit410 stores the camera image and the range image in the current frame asa preceding-frame camera image and a preceding-frame range image,respectively, before returning to S821. When the determination isaffirmative in S826 (YES), the object detection unit 410 proceeds toS827.

In S827, the object detection unit 410 calculates a difference imagebetween a range image in a latest frame and the document standbackground range image and performs a labeling process on the differenceimage. Thereafter, the object detection unit 410 determines that a labelhaving a predetermined number of pixels or more corresponds to a regionwhere an object is placed, and records a rectangle region including thepixels having the label as object placement region information of theplaced object. When a plurality of labels have a predetermined number ofpixels or more, the object detection unit 410 determines that aplurality of objects have been placed and records individual objectplacement region information of the objects. After S827, the objectdetection unit 410 proceeds to S829.

In S829, the object detection unit 410 stores the current frame rangeimage as a “preceding-object-placement range image”. Thereafter, theobject detection unit 410 notifies the main controller 402 of theplacement of the object in addition to the object placement regioninformation (an object placement notification) and terminates the objectplacement detection process of FIG. 5B.

FIG. 5C is a detailed flowchart of the object placement detectionprocess in S802 of FIG. 5A performed when a second object onwards areplaced. In FIG. 5C, processes the same as those in FIG. 5B are denotedby reference numerals the same as those in FIG. 5B, and descriptionsthereof are omitted.

After S821 of FIG. 5C, the object detection unit 410 proceeds to S841.In S841, the object detection unit 410 calculates a difference betweenan obtained range image and the “preceding-object-placement range image”and calculates a difference value by adding absolute values of thedifference. After S841, the object detection unit 410 proceeds to S842.

In S842, the object detection unit 410 determines whether the calculateddifference value is equal to or larger than a predetermined value. Whenthe determination is negative in S842 (NO), the object detection unit410 determines that a new object is not captured and proceeds to S828.On the other hand, when the determination is affirmative in S842 (YES),the object detection unit 410 determines that a new object is capturedand proceeds to S824.

Furthermore, when determining that a predetermined number of consecutiveframes have difference values which are equal to or smaller than apredetermined value (YES) in S826 of FIG. 5C, the object detection unit410 proceeds to S847 in FIG. 5C. In S847, the object detection unit 410calculates a difference image between a range image in a latest frameand the “preceding-object-placement range image” and performs a labelingprocess on the difference image. Note that the process in S827 isperformed on a result of the labeling. After S847, the object detectionunit 410 proceeds to S829.

FIG. 5D is a flowchart of the object removal detection process in S803of FIG. 5A illustrated in detail.

When starting the object removal detection process, the object detectionunit 410 obtains a range image for one frame from the range imageobtaining unit 408 in S831 of FIG. 5D. After S831, the object detectionunit 410 proceeds to S832.

In S832, the object detection unit 410 calculates a difference valuebetween the obtained range image and the document stand background rangeimage. After S832, the object detection unit 410 proceeds to S833.

In S833, the object detection unit 410 determines whether the calculateddifference value is equal to or smaller than a predetermined value. Whenthe determination is negative (NO) in S833, the object detection unit410 returns to S831 so as to continue the process since an object isstill on the document stand 204. On the other hand, when thedetermination is affirmative in S823 (YES), the process proceeds toS834. Thereafter, since an object is removed from the document stand204, the object detection unit 410 transmits a notification (an objectremoval notification) to the main controller 402 and terminates theobject removal detection process of FIG. 5C.

Hardware Configuration of Controller of Smartphone

FIG. 6 is a diagram schematically illustrating a hardware configurationof a controller unit 801 of the smartphone 106 of FIG. 1.

The controller unit 801 of FIG. 6 includes a CPU 902, a RAM 903, a ROM904, a flash memory 905, a network I/F 906, a display controller 909,and a serial I/F 910. These units are connected to a system bus 901.

The CPU 902 controls an entire operation of the controller unit 801. TheRAM 903 is a volatile memory. The ROM 904 is a nonvolatile memory whichstores a boot program of the CPU 902. The flash memory 905 is anonvolatile memory having larger capacity than the ROM 904. The flashmemory 905 stores a control program of the smartphone 106 to be executedby the controller unit 801.

The CPU 902 executes the boot program stored in the ROM 904 at a time ofactivation, such as power-ON. When the boot program is executed, thecontrol program stored in the flash memory 905 is read and developed theRAM 903. After executing the boot program, the CPU 902 subsequentlyexecutes the control program developed in the RAM 903 so as to controlthe various units. Furthermore, the CPU 902 stores data to be used foran operation based on the control program in the RAM 903 and performsreading and writing. The flash memory 905 may further store varioussettings required for the operation based on the control program andvarious data, including image data which are read and written undercontrol of the CPU 902. The CPU 902 performs communication with otherdevices on the network 104 through the network I/F 906.

The display controller 909 is connected to a display of an LCD touchpanel 990. The display controller 909 controls the display of images ona display of the LCD touch panel 990 in accordance with an instructionissued by the CPU 902.

The serial I/F 910 inputs and outputs a serial signal. Here, the serialI/F 910 is connected to the LCD touch panel 990. When the LCD touchpanel 990 is pressed, the CPU 902 obtains a coordinate of a pressedposition through the serial I/F 910.

Concrete Example of Data Transfer Between Smartphone and Camera Scanner

Hereinafter, an example of transfer of image data from the smartphone106 to the camera scanner 101, which displays the image data, will bedescribed in detail with reference to FIGS. 7A to 7G, FIGS. 8A to 8H,and FIGS. 9A and 9B.

FIGS. 7A to 7G are diagrams illustrating screen transitions of thesmartphone 106 when a data transfer process is performed.

FIGS. 8A to 8H are diagrams illustrating states of the surface of thedocument stand 204 of the camera scanner 101 when the data transferprocess is performed, that is, an image projected by the projector 207and an object physically placed on the document stand 204 aresimultaneously displayed.

FIG. 8A is a diagram illustrating a screen projected by the projector207 and displayed on the document stand 204 immediately after the camerascanner 101 is activated. The projection screen, which is displayedimmediately after the activation that is illustrated in FIG. 8A displaysa scan button 1101 to be operated by the user when scanning is performedby the camera scanner 101 and a smartphone data display button 1102 tobe operated by the user when an image transferred from the smartphone106 is displayed. The process of transferring an image from thesmartphone 106 to the camera scanner 101 is started when the useroperates the smartphone data display button 1102 on the projectionscreen of the camera scanner 101 illustrated in FIG. 8A. Specifically,the main controller 402 of the camera scanner 101 starts a processingflow of FIG. 9B, described below, using the user operation as a trigger.

Furthermore, the CPU 902 of the smartphone 106 starts the process ofimage transfer when the user performs the following operation.

First, when the user inputs an operation instruction for displaying ahome screen for image transfer, the CPU 902 of the smartphone 106 causesa display to display the home screen illustrated in FIG. 7A, andthereafter activates image viewer application software. The applicationsoftware is simply referred to as “app” hereinafter.

Furthermore, the CPU 902 displays a list of thumbnail images asillustrated in FIG. 7B on a display screen of the LCD touch panel 990through the display controller 909. Although not described hereinafter,the display on the display screen of the LCD touch panel 990 isperformed through the display controller 909 under control of the CPU902.

When the user selects a thumbnail image 1002 of an image to betransferred from the list of thumbnail images illustrated in FIG. 7B,the CPU 902 displays an original image 1003 of the selected thumbnailimage 1002 as illustrated in FIG. 7C. Furthermore, the CPU 902 alsodisplays a “share” button 1004 at this time. Note that, although notillustrated, the selection by the user on the screen is performedthrough a touch operation on the LCD touch panel 990. The CPU 902recognizes the selection of the user based on information on theoperation performed by the user on the LCD touch panel 990.

When the user selects the share button 1004, the CPU 902 displays asharable app list as illustrated in FIG. 7D. Here, the CPU 902 displaysan image transfer app stored in the flash memory 905 in advance in thesharable app list. The image transfer app is used to execute a processin FIG. 9A described below.

Subsequently, when the user selects an item 1005 of a camera scanner appin the sharable app list, for example, the CPU 902 starts a processingflow of FIG. 9A described below. Thereafter, the user places thesmartphone 106 on a smartphone placement region 1103, described below,on the document stand 204.

FIG. 9A is a flowchart of the data transfer process performed by thesmartphone 106. A process of the flowchart in FIG. 9A is executed by theCPU 902 of the smartphone 106 and started when the user selects one ofthe items 1005 of the camera scanner app in the sharable app list ofFIG. 7D.

In S1201 of FIG. 9A, the CPU 902 sets a unique identifier to aphotograph to be transmitted (corresponding to an image selected in FIG.7B) and stores the photograph with the unique identifier in the RAM 903.Furthermore, the CPU 902 encodes data transfer information including theidentifier of the photograph and connection information required forconnection of an external apparatus (the camera scanner 101 in thisembodiment) to the smartphone 106 by a predetermined encoding method soas to generate an encoded image. In this embodiment, an IP address ofthe smartphone 106 is used as the connection information. Specifically,the CPU 902 generates a character string (1: 192.268.0.1, for example)described in a format “<an identifier of a photograph to betransmitted>:<an IP address of the smartphone 106>” as data transferinformation. Then the CPU 902 encodes the character string of the datatransfer information to obtain a QR code as an example of an imageencoded by the predetermined encoding method. Furthermore, the CPU 902displays an image of the QR code in a display of the LCD touch panel 990as illustrated in FIG. 7E. Thereafter, the QR code displayed in thedisplay of the LCD touch panel 990 of the smartphone 106 placed on thedocument stand 204 is captured and decoded (decrypted) by the camerascanner 101 so that the camera scanner 101 obtains data transferinformation. Note that the data transfer information may be encoded byanother method instead of the QR code as long as the camera scanner 101may detect (decode) the data transfer information. Although theidentifier of the photograph in the data transfer information isrepresented by a single digit number for simplicity, the information maybe represented by another expression method. Note that, when theidentifier is short, data to be transferred to the camera scanner 101 bythe user may be obtained by a third person who attempts to access theidentifier by a brute-force attack. Therefore, the identifier preferablyhas an efficiently long identifier name in terms of security. AfterS1201, the CPU 902 proceeds to S1202.

In S1202, the CPU 902 opens a socket in a predetermined port (12345, forexample) and enters a connection waiting state. Thereafter, when a datatransfer request of a format “SENDREQ: <identifier of photograph>”transmitted in S1227 of FIG. 9B described below is received by thisport, the CPU 902 returns “ACK” and proceeds to S1203.

In S1203, the CPU 902 compares the identifier of the photograph includedin the data transfer request received in S1202 with the identifier ofthe photograph set in S1201. If the identifiers match each other, theCPU 902 starts transfer of data of the photograph, and in addition,displays a screen indicating that the data transfer is being performed,for example as illustrated in FIG. 7F. Thereafter, when the datatransfer is completed, the CPU 902 proceeds to S1204.

In S1204, the CPU 902 displays a screen indicating that the datatransfer is completed in the display, for example, as illustrated inFIG. 7G.

FIG. 9B is a flowchart of the data transfer process performed by thecamera scanner 101. A flow of the data transfer process is executed whenthe main controller 402 of the camera scanner 101 controls the variousunits and is started when the initializing process in S801 of FIG. 5A,described above, is terminated.

In S1221 of FIG. 9B, the main controller 402 causes the projector 207 toproject and display a screen that prompts the user to place thesmartphone 106 on the document stand 204. For example, the screenillustrated in FIG. 8B may be projected and displayed on the documentstand 204. The screen illustrated in FIG. 8B includes the smartphoneplacement region 1103. Although a description is omitted hereinafter,the projection and display on the document stand 204 are performed undercontrol of the main controller 402 through the user interface unit 403and the display unit 406. After S1221, the main controller 402 proceedsto S1222.

In S1222, the main controller 402 causes the object detection unit 110to start the object detection process. After S1222, the main controller402 proceeds to S1223.

In S1223, the main controller 402 waits until the object detection unit410 detects an object placed on the document stand 204 and anotification indicating the detection is supplied from the objectdetection unit 410. When receiving the notification indicating thedetection of the object placement in S1223, the main controller 402proceeds to S1224. In a case of this embodiment, a state of the documentstand 204 is illustrated in FIG. 8C at this time point. Specifically,the document stand 201 is in a state in which the smartphone 106 thatdisplays the QR code on the screen of the smartphone 106 is placed onthe document stand 204.

In S1224, the main controller 402 obtains a camera image from the cameraimage obtaining unit 407, extracts a region corresponding to the objectin the camera image based on the object placement region information,and decodes the QR code included in the extracted region by a generalmethod. Note that, when the camera image is to be obtained, a blackimage may be displayed on the entire surface of the document stand 204in advance (the entire screen is filled with black) so that reflectionof display of the GUI components do not degrade detection accuracy ofthe QR code, and the original display may be obtained after the cameraimage is obtained. After S1224, the main controller 402 proceeds toS1225.

In S1225, the main controller 402 determines whether information on aformat “<identifier of photograph>:<IP address>” (text information, forexample) is obtained as a result of the decoding of the QR code. Whenthe determination is affirmative (YES) in S1225, the main controller 402proceeds to S1227. When the determination is negative (NO) in S1225, themain controller 402 proceeds to S1226.

In S1226, the main controller 402 projects and displays an error screen,which prompts the user to operate a smartphone app. The state of thedocument stand 204 is illustrated in FIG. 8G at this time point. AfterS1226, the main controller 402 returns to S1223.

A process in S1227 is performed by the data transfer processing unit 420under control of the main controller 402. In S1227, the data transferprocessing unit 420 transmits a request for transferring data of aformat “SENDREQ:<identifier of photograph>” to the predetermined port(12345, for example) of the IP address obtained in S1224. The identifierof the photograph transmitted here is obtained in S1224. After S1227,the data transfer processing unit 420 proceeds to S1228.

S1228, the data transfer processing unit 420 determines whether aresponse “ACK” has been transmitted from the smartphone 106. When thedetermination is affirmative in S1228 (YES), the main controller 402proceeds to S1230. When the determination is negative in S1228 (NO), themain controller 402 proceeds to S1229.

In S1229, the main controller 402 projects and displays an error screenwhich prompts the user to check a network environment. The state of thedocument stand 204 is illustrated in FIG. 8H at this time point. AfterS1229, the main controller 402 returns to S1223.

In S1230, the main controller 402 projects and displays a data receptionstart screen, which notifies the user of a reception of data. The stateof the document stand 204 is illustrated in FIG. 8D at this time point.After S1230, the main controller 402 proceeds to S1231.

A process in S1231 is performed by the data transfer processing unit 420under control of the main controller 402. In S1231, the data transferprocessing unit 420 continues the data reception process until thereception of data transmitted from the smartphone 106 is completed. Whenthe reception of data transmitted from the smartphone 106 is completed(completion of data transfer), the data transfer processing unit 420notifies the main controller 402 of the completion and the maincontroller 402 proceeds to S1232.

In S1232, the main controller 402 projects and displays a data transfercompletion screen, which notifies the user of the completion of datatransfer. The image 1003 of the photograph received from the smartphone106 is displayed in the projected and displayed screen. Here, if themain controller 402 does not receive a notification indicating that theobject has been removed from the document stand 204 from the objectdetection unit 410 before the process in S1232, the main controller 402displays the image 1003 of the photograph in a region different from thesmartphone placement region 1103. The document stand 204 enters a stateof FIG. 8E, and the image 1003 of the photograph is displayed in anupper portion of the screen, for example. On the other hand, if the maincontroller 402 has received the notification indicating that the objecthas been removed from the document stand 204 from the object detectionunit 410 before the process in S1232, the main controller 402 displaysthe image 1003 of the photograph in the center of the screen.Furthermore, the main controller 402 also displays an end button 1104,which receives an instruction for terminating the screen display fromthe user. The state of the document stand. 204 is illustrated in FIG. 8Fat this time point. The end button 1104 is operated when the userinstructs an end of the data display app for displaying data transferredfrom the smartphone 106. When the end button 1104 is pressed by theuser, the main controller 402 returns to an initial screen (FIG. 8A).

As described above, according to the first embodiment, the user mayperform data transfer from the smartphone 106 to the camera scanner 101by a small number of operations. In particular, if the operation isstarted in a state in which the user desires transfer of an image to acamera scanner while viewing the image on the screen of the smartphone106, the user only requires the following operations.

The share button 1004 is pressed in the state of FIG. 7C.

One of the items 1005 of the camera scanner 101 is selected in the stateof FIG. 7D.

The smartphone data display button 1102 in FIG. 8A is pressed.

The smartphone is placed on the document stand 204 in the state of FIG.8B.

Although the transferred data is photograph data in the exampledescribed above, the data is not limited to photograph data, andmoving-image data or text data may be transferred. Furthermore, althoughthe operation of pressing the smartphone data display button 1102 may berequired on the camera scanner 101 side, this operation may be omitted.The camera scanner 101 may constantly recognize a QR code. By this, oneof the operations required to be performed by the user described above,that is, the operation indicated by “the smartphone data display button1102 is pressed in FIG. 8A” is not required.

Second Embodiment

Although the communication between the camera scanner 101 and the singlesmartphone 106 is described as an example in the first embodiment,communication between a camera scanner 101 and a plurality ofsmartphones 106 will be described as an example in a second embodiment.Note that the configurations of the camera scanner 101 and thesmartphones 106 are the same as those described above, and therefore,illustrations and descriptions thereof are omitted.

The data transfer process according to the second embodiment will bedescribed hereinafter with reference to FIGS. 10A to 10K and FIG. 11.

FIGS. 10A to 10K are diagrams illustrating a state of a surface of adocument stand 204 of the camera scanner 101 when the data transferprocess is performed in the second embodiment. In the second embodiment,a user is not required to press a smartphone data display button 1102 inan initial display screen illustrated in FIG. 10A, which is similar toFIG. 8A described above. It is assumed here That the user presses(selects) the scan button 1101, for example. When the user selects thescan button 1101, the camera scanner 101 activates a scan app anddisplays a scan app screen, for example, as illustrated in FIG. 10B.Note that the activation of the scan app may not be required in order torealize a data transfer process described below, and the initial screendisplayed before the scan button 1101 or the like is selected may becontinuously displayed.

FIG. 11 is a flowchart of the data transfer process performed by acamera scanner 101 according to the second embodiment. The flow of thedata transfer process of FIG. 11 is executed when a main controller 402of the camera scanner 101 controls various units and is started whenactivation of the camera scanner 101 is completed. Note that, in theflowchart of FIG. 11, the processes the same as those of FIG. 9Bdescribed above are denoted by reference numerals the same as those ofFIG. 9B, and descriptions thereof are omitted. Furthermore, processesperformed by a smartphone 106 are the same as those described in thefirst embodiment, and therefore, descriptions thereof are also omitted.

In a case of the flowchart of FIG. 11, after S1222, the main controller402 proceeds to S1223, and then to S1224. After S1224, the maincontroller 402 proceeds to S1425.

In S1425, the main controller 402 determines whether data of a format“<identifier of photograph>:<IP address>” (text data) has been obtainedas a result of decoding a QR code in S1224. When the determination isaffirmative (YES) in S1425, the main controller 402 proceeds to S1601.On the other hand, when the determination is negative (NO) in S1425, themain controller 402 returns to S1223. Furthermore, also in a case wherethe data of the format “<identifier of photograph>:<IP address>” is notobtained in S1245 because no QR code is detected in S1224, the maincontroller 402 returns to S1223. Although the error display is performedin S1225 when the data of the format “<identifier of photograph>:<IPaddress>” is not obtained according to the foregoing first embodiment,the error display is not performed in the second embodiment. This isbecause, if the error display is performed, an error is constantlydisplayed in a case where an object other than smartphones is placed,such as a case where a paper document is to be simply scanned, forexample.

In S1601, the main controller 402 performs a reception data displayregion determination process, which is described below. The receptiondata display region determination process determines a reception datadisplay region for displaying data or a message associated with thesmartphone 106 placed on the document stand 204. The reception datadisplay region is represented as rectangle regions denoted by dottedlines in FIGS. 10D to 10K (a reception data display region 1501 of FIG.10D, for example). After S1601, the main controller 402 proceeds toS1602.

In S1602, the main controller 402 determines whether the reception datadisplay region has been ensured in S1601. When the determination isnegative in S1602 (NO), the main controller 402 returns to S1223. On theother hand, when the determination is affirmative in S1602 (YES), themain controller 402 proceeds to S1227.

Thereafter, the process proceeds until S1228 of FIG. 11, and when it isdetermined that a response “ACK” has been supplied from the smartphone106 in S1228 (YES), the main controller 402 proceeds to S1630. On theother hand, when it is determined that a response “ACK” has not beensupplied from the smartphone 106 in S1228 (NO), the main controller 402proceeds to S1629.

In S1629, the main controller 402 displays an error screen that promptsthe user to check a network environment in the reception data displayregion obtained in S1601. Thereafter, the main controller 402 returns toS1223.

On the other hand, in S1630, the main controller 402 displays a datareception start message that notifies a user of a reception of data inthe reception data display region 1501 determined in S1601, for example,as illustrated in FIG. 10D. Furthermore, when an end button 1104 isbeing displayed, the main controller 402 does not display the datareception start message. After S1630, the main controller 402 proceedsto S1231.

In S1231, when receiving a notification indicating that the datatransfer is completed from the data transfer processing unit 420, themain controller 402 proceeds to S1632.

In S1632, the main controller 402 projects and displays a data transfercompletion screen that notifies the user of the completion of datatransfer. Here, the main controller 402 projects and displays the screensuch that a direction of the QR code obtained in S1224 matches adirection of the displayed photograph. In this way, the photograph isdisplayed in an appropriate direction when viewed from the user whoplaced the smartphone 106. Furthermore, in addition to the display ofthe received photograph, the end button 1104 is disposed on this screenso as to accept an end of display. As with the case of the firstembodiment, the end button 1104 is selected when the user instructs anend of the data transfer completion screen. When the end button 1104 ispressed, the main controller 402 changes a screen so that the screendisplayed before the data reception start screen displayed in S1230 isdisplayed. Furthermore, the main controller 402 clears all set receptiondata display regions. Note that end buttons 1104 are not disposed fordifferent reception data display regions, but only one end button 1104is displayed for the entire screen. After S1632, the main controller 402proceeds to S1433.

In S1433, the main controller 402 determines whether the user hasrequested shutdown of an entire system. When the determination isaffirmative in S1433 (YES), the main controller 402 terminates theprocess in the flowchart of FIG. 11 and executes the shutdown of thesystem. On the other hand, when the determination is negative in S1433(NO), the main controller 402 returns to S1223. Specifically, the datatransfer process of the second embodiment is repeatedly performed untilthe entire system is shut down.

Note that, although all the processes are performed in synchronizationwith one another for simplicity in this embodiment, a long period oftime may be required for a reception of data in practice, and therefore,the processes in S1231 and S1632 may not be synchronized with each otherso that data receptions from a plurality of smartphones aresimultaneously performed. Specifically, after the process in S1630 isexecuted, the process in S1433 is entered, and simultaneously, theprocesses in S1231 and S1632 are executed in turn in an asynchronousmanner. Furthermore, when a plurality of objects are simultaneouslydetected in S1223, the process in S1224 onwards is performed for eachobject region.

Hereinafter, the relationships between the processes in the steps in thedata transfer process according to the second embodiment and the stateof the document stand 204 will be described.

First, it is assumed that a first smartphone 106 is placed on thedocument stand 204 as illustrated in FIG. 10C. In the second embodiment,a state of the document stand 204 at a time when the process in S1630 ofFIG. 11 is performed is illustrated in FIG. 10D. As illustrated in FIG.10D, information associated with the first smartphone 106 is displayedin the reception data display region 1501. Furthermore, a state of thedocument stand 204 at a time when the process in S1629 of FIG. 11 isperformed is illustrated in FIG. 10K.

Furthermore, a state of the document stand 204 at a time when theprocess in S1632 of FIG. 11 is performed is illustrated in FIG. 10E in acase where the main controller 402 has not received an object removalnotification from an object detection unit 410 before the process inS1632 is performed. On the other hand, a state of the document stand 204in a case where the main controller 402 has received an object removalnotification from the object detection unit 410 is illustrated in FIG.10F.

Subsequently, it is assumed that a second smartphone is placed on thedocument stand 204, for example, as illustrated in FIG. 10G in S1223 ofFIG. 11, while the first smartphone 106 is still placed on the documentstand 204. In this case, a state of the document stand 204 at a timewhen the process in S1630 of FIG. 11 is performed for the second time isillustrated in FIG. 10H.

Furthermore, a state of the document stand 204 at a time when theprocess in S1632 is performed for the second time in a case where themain controller 402 has not received an object removal notification fromthe object detection unit 410 before the process in S1632 is performedis illustrated in FIG. 10I. On the other hand, a state of the documentstand 204 in a case where the main controller 402 has received an objectremoval notification from the object detection unit 410 is illustratedin FIG. 10J.

FIG. 12 is a flowchart of the reception data display regiondetermination process in S1601 of FIG. 11 illustrated in detail.

In S1701 of FIG. 12, the main controller 402 obtains object placementregion information. After S1701, the main controller 402 proceeds toS1702.

In S1702, the main controller 402 sets rectangle regions having apredetermined size which accommodate an object placement region, thatis, a rectangle region having a longer side extending upward, arectangle region having a longer side extending rightward, a rectangleregion having a longer side extending downward, and a rectangle regionhaving a longer side extending leftward, relative to the objectplacement region serving as a reference. Then the main controller 402determines whether each of the rectangle regions protrudes outside theentire display region and whether each of the rectangle regions overlapswith another reception data display region that has been set so as todetermine whether a blank region exists. After S1702, the maincontroller 402 proceeds to S1703.

In S1703, the main controller 402 determines whether a blank regionexists. When the determination is negative in S1703 (NO), the maincontroller 402 proceeds to S1704, and otherwise (YES), the maincontroller 402 proceeds to S1705.

In S1704, the main controller 402 displays an error message indicatingthat an image of data which has been transferred may not be displayed ina position where the object (the smartphone 106) is placed by the user,and thereafter, the main controller 402 terminates the process in FIG.12 and proceeds to S1602 of FIG. 11.

On the other hand, in S1705, the main controller 402 sets the rectangleregion obtained in S1702 as a reception data display region of theplaced object (the smartphone 106), and thereafter, the main controller402 terminates the flowchart of FIG. 12 and proceeds to S1602 of FIG.11.

FIGS. 13A to 13E are diagrams illustrating states of the document stand204 when the reception data display region determination processillustrated in FIG. 12 is executed. In FIGS. 13A to 13E, rectangleregions surrounded by dotted lines indicate reception data displayregions.

FIG. 13A is a diagram illustrating a state in which a first smartphone106 is placed on the document stand 204. In an example of FIG. 13A,since a rectangle region having a longer side extending leftwardrelative to the smartphone 106 may be set, the rectangle region whichextending leftward is set as the reception data display region.

FIG. 13B is a diagram illustrating a state in which a second smartphoneis placed on the document stand 204. In an example of FIG. 13B, arectangle region having a longer side extending leftward, upward, ordownward may not be set, and therefore, a rectangle region extendingrightward is set as the reception data display region. FIG. 13C is adiagram illustrating a state in which a third smartphone is placed onthe document stand 204. In an example of FIG. 13C, since a rectangleregion having a longer side extending leftward, upward, or rightward maynot be set, and therefore, a rectangle region which extending downwardis set as the reception data display region. FIG. 13D is a diagramillustrating a state in which a fourth smartphone is placed on thedocument stand 204. In an example of FIG. 13D, since a rectangle regionhaving a longer side extending leftward, upward, rightward, or downwardmay not be set, and therefore, an error message is displayed asillustrated in. FIG. 13E.

As described above, according to the second embodiment, even in a casewhere photograph data is transferred from the plurality of smartphones106 to the camera scanner 101, the user may perform data transfer fromthe smartphones 106 to the camera scanner 101 by a smaller number ofoperations. Furthermore, according to the second embodiment, datatransferred from each of the smartphones 106 is displayed in thevicinity of a corresponding one of the smartphones 106 placed on thedocument stand 204 that has transferred the data. Accordingly, the usersof the smartphones 106 may check the transferred data on the documentstand 204 without confusion. Note that, although the transferredphotographs are displayed in the vicinity of the correspondingsmartphones 106 that have transferred data of the photographs in thisembodiment, the relationships between the photographs and thesmartphones 106 may be emphasized by connecting the smartphones 106 tothe photographs by images of lines, for example.

As described above, according to the first and second embodiments, dataincluded in a mobile terminal, such as a tablet device or a smartphone,may be immediately transferred to a camera scanner and displayed in alarge screen. Furthermore, the series of operations for the datatransfer includes an operation of placing the device of a transfersource on a document stand of the camera scanner, which is a transferdestination, and therefore, the data transfer is realized by anoperation that is simple for a user. Therefore, according to the firstand second embodiments, efficiency of operations using the camerascanner may be improved.

Other Embodiments

Some embodiment(s) can be realized by a computer of a system orapparatus that reads out and executes computer-executable instructions(e.g., one or more programs) recorded on a storage medium (which mayalso be referred to more fully as a ‘non-transitory computer-readablestorage medium’) to perform the functions of one or more of theabove-described embodiment(s) and/or that includes one or more circuits(e.g., an application specific integrated circuit (ASIC)) for performingthe functions of one or more of the above-described embodiment(s), andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer-executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s) and/or controlling the one or morecircuits to perform the functions of one or more of the above-describedembodiment(s). The computer may comprise one or more processors (e.g.,central processing unit (CPU), micro processing unit (MPU)) and mayinclude a network of separate computers or separate processors to readout and execute the computer-executable instructions. Thecomputer-executable instructions may be provided to the computer, forexample, from a network or the storage medium. The storage medium mayinclude, for example, one or more of a hard disk, a random-access memory(RAM), a read only memory (ROM), a storage of distributed computingsystems, an optical disk (such as a compact disc (CD), digital versatiledisc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memorycard, and the like.

While the present disclosure describes exemplary embodiments, it is tobe understood that the embodiments are not limited to the disclosedexemplary embodiments. Also, the scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims priority to Japanese Patent Application No.2016-218954 filed Nov. 9, 2016, which is hereby incorporated byreference in its entirety.

What is claimed is:
 1. An image processing system in which a mobileterminal and an image processing apparatus are able to perform wirelesscommunication with each other, the mobile terminal comprising: a displaycapable of displaying information; a storage capable of storing imagedata; and a controller capable of causing the display to display anencoded image which is obtained by encoding both of connectioninformation and identification information, wherein the connectioninformation is information for specifying the mobile terminal, andwherein the identification information is information for identifyingimage data that is selected from images stored in the mobile terminal bya user of the mobile terminal; and the image processing apparatuscomprising: a projector capable of projecting an image onto a projectionplane; an imaging unit capable of capturing the projection plane toobtain a captured image; and at least one controller configured toperform processing, including acquiring, from the captured imageobtained by the imaging unit, the encoded image displayed on the mobileterminal placed on the projection plane, decoding the acquired encodedimage to obtain both of the connection information and theidentification information, requesting the mobile terminal specifiedwith the decoded connection information, to transfer the image dataidentified by the decoded identification information, receiving, fromthe mobile terminal, the image data identified by the decodedidentification information, and causing the projector to project animage based on the received image data, wherein the projector of theimage processing apparatus is capable of projecting the image based onthe received image data in a state in which the mobile terminal has beenremoved from the projection plane after acquisition of the encodedimage.
 2. The information processing system according to claim 1,wherein the projector of the image processing apparatus projects, ontothe projection plane, information about an area where the mobileterminal is to be placed.
 3. The information processing system accordingto claim 1, wherein the projector of the image processing apparatusprojects, onto the projection plane, an object for giving an instructionfor ending the projection of the image based on the received image data.4. The information processing system according to claim 1, wherein theprojector of the image processing apparatus projects, onto theprojection plane, information that indicates a status of communicationwith the mobile terminal when receiving the image data.
 5. Theinformation processing system according to claim 1, wherein theprojector of the image processing apparatus projects the image based onthe received image data in such a way as not to overlap a position wherethe mobile terminal is placed.
 6. The information processing systemaccording to claim 1, wherein the projector of the image processingapparatus projects the image based on the received image data at aposition based on a read position of the encoded image.
 7. Theinformation processing system according to claim 1, wherein the imageprocessing apparatus performs first projection processing based on thereceived image data in a state in which the mobile terminal is placed onthe projection plane and performs second projection processing based onthe received image data in a state in which the mobile terminal has beenremoved from the projection plane.
 8. The information processing systemaccording to claim 7, wherein a projection size of the received imagedata projected in the second projection processing is larger than aprojection size of the received image data projected in the firstprojection processing.
 9. The information processing system according toclaim 1, wherein the image processing apparatus performs errornotification based on a placement, in a predetermined situation, of themobile terminal that displays the encoded image.
 10. The imageprocessing system according to claim 9, wherein the predeterminedsituation is that an area for further projecting another image is notavailable on the projection plane.
 11. The information processing systemaccording to claim 1, wherein the image processing apparatus makes anarea for projecting the image based on the received image data availablein response to a placement, on the projection plane, of the mobileterminal that displays the encoded image.
 12. The information processingsystem according to claim 1, wherein, after receiving the request fromthe image processing apparatus, the mobile terminal executes screenswitching automatically from a screen with the encoded image to a screenwithout the encoded image.
 13. The information processing systemaccording to claim 12, wherein the screen without the encoded image is ascreen with a message that the mobile terminal be removed from theprojection plane.
 14. The image processing system according to claim 1,wherein the connection information includes an IP address of the mobileterminal.
 15. The image processing system according to claim 1, whereinthe encoded image is in a barcode format.
 16. The image processingsystem according to claim 15, wherein the barcode format is a QR codeformat.
 17. An image processing apparatus capable of performing wirelesscommunication with a mobile terminal, the image processing apparatuscomprising: a projector capable of projecting an image onto a projectionplane; an imaging unit capable of capturing the projection plane toobtain a captured image; and at least one controller configured toperform processing, including acquiring, from the captured imageobtained by the imaging unit, an encoded image displayed on the mobileterminal placed on the projection plane, decoding the acquired encodedimage to obtain both of connection information and identificationinformation, requesting the mobile terminal specified with the decodedconnection information to transfer image data identified by the decodedidentification information, receiving, from the mobile terminal, theimage data identified by the decoded identification information, andcausing the projector to project an image based on the received imagedata, wherein the projector is capable of projecting the image based onthe received image data in a state in which the mobile terminal has beenremoved from the projection plane after acquisition of the encodedimage.
 18. A method for controlling an image processing apparatuscapable of performing wireless communication with a mobile terminal andincluding a projector capable of projecting an image onto a projectionplane and an imaging unit capable of capturing the projection plane toobtain a captured image, the method comprising: acquiring, from thecaptured image obtained by the imaging unit, an encoded image displayedon the mobile terminal placed on the projection plane; decoding theacquired encoded image to obtain both of connection information andidentification information; requesting the mobile terminal specifiedwith the decoded connection information to transfer image dataidentified by the decoded identification information; receiving, fromthe mobile terminal, the image data identified by the decodedidentification information; and causing the projector to project animage based on the received image data, wherein the projector is capableof projecting the image based on the received image data in a state inwhich the mobile terminal has been removed from the projection planeafter acquisition of the encoded image.
 19. An image processing systemin which a mobile terminal and an image processing apparatus are able toperform wireless communication with each other, the mobile terminalcomprising: a display capable of displaying information; a storagecapable of storing image data; and a controller capable of causing thedisplay to display an encoded image which is obtained by encoding bothof connection information and identification information, wherein theconnection information is information for specifying the mobileterminal, and wherein the identification information is information foridentifying image data that is selected from images stored in the mobileterminal by a user of the mobile terminal; and the image processingapparatus comprising: a projector capable of projecting an image onto aprojection plane; an imaging unit capable of capturing the projectionplane to obtain a captured image; and at least one controller configuredto perform processing, including acquiring, from the captured imageobtained by the imaging unit, the encoded image displayed on the mobileterminal placed on the projection plane, decoding the acquired encodedimage to obtain both of the connection information and theidentification information, requesting the mobile terminal specifiedwith the decoded connection information, to transfer the image dataidentified by the decoded identification information, receiving, fromthe mobile terminal, the image data identified by the decodedidentification information, and causing the projector to project animage based on the received image data, wherein the projector is capableof projecting the image based on the received image data in a state inwhich the mobile terminal on the projection plane does not display theencoded image after receiving the request from the image processingapparatus.
 20. An image processing apparatus capable of performingwireless communication with an electronic terminal, the image processingapparatus comprising: a projector capable of projecting an image onto aprojection plane; an imaging unit capable of capturing the projectionplane to obtain a captured image; and at least one controller configuredto perform processing, including acquiring, from the captured imageobtained by the imaging unit, an encoded image displayed on theelectronic terminal placed on the document stand, decoding the acquiredencoded image to obtain both of connection information andidentification information, requesting the mobile terminal specifiedwith the decoded connection information, to transfer image dataidentified by the decoded identification information, receiving, fromthe mobile terminal, the image data identified by the decodedidentification information, and causing the projector to project animage based on the received image data, wherein the projector is capableof projecting the image based on the received image data in a state inwhich the mobile terminal does not display the encoded image afterreceiving the request from the image processing apparatus.